Reachability Analysis with State-Compatible Automata

Regular tree languages are a popular device for reachability analysis over term rewrite systems, with many applications like analysis of cryptographic protocols, or confluence and termination analysis. At the heart of this approach lies tree automata completion, first introduced by Genet for left-linear rewrite systems. Korp and Middeldorp introduced so-called quasi-deterministic automata to extend the technique to nonleft-linear systems. In this paper, we introduce the simpler notion of state-compatible automata, which are slightly more general than quasideterministic, compatible automata. This notion also allows us to decide whether a regular tree language is closed under rewriting, a problem which was not known to be decidable before. Several of our results have been formalized in the theorem prover Isabelle/HOL. This allows to certify automatically generated non-confluence and termination proofs that are using tree automata techniques.